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Abstract. A significant challenge in nature-inspired algorithmics is the 
identification of specific characteristics of problems that make them harder 
(or easier) to solve using specific methods. The hope is that, by identi- 
fying these characteristics, we may more easily predict which algorithms 
are best-suited to problems sharing certain features. Here, we approach 
this problem using fitness landscape analysis. Techniques already exist 
for measuring the "difficulty" of specific landscapes, but these are often 
designed solely with evolutionary algorithms in mind, and are generally 
specific to discrete optimisation. In this paper we develop an approach 
for comparing a wide range of continuous optimisation algorithms. Us- 
ing a fitness landscape generation technique, we compare six different 
nature-inspired algorithms and identify which methods perform best on 
landscapes exhibiting specific features. 

1 Introduction 

Inspired by the foundational work of Wolpert and Macready [l] , practitioners 
have long sought to better understand the relationship between problems and 
solution methods (i.e., algorithms). Here, we are particularly interested in the 
question of which algorithm is best-suited to a particular problem, and the process 
of addressing this has been described by some as a "black-art" 

Although theoretical studies in this area have yielded useful results, the ex- 
perimental analysis of algorithms is receiving increasing attention. As Morgan 
and Gallagher point out [sl, this approach is scalable in that it readily admits 
newly-described algorithms, and it is now an area of research that is supported 
by a number of high-profile competitions and libraries of benchmark test prob- 
lems. 

The fundamental properties of a problem's search landscape underpin much 
work in experimental analysis, and the use of landscape/test case generators [3||7j 
has been proposed as one way in which we might effectively assess algorithms 
against problem instances. 

In this paper we examine six different nature-inspired algorithms by testing 
them against a number of different randomized landscapes with several different 
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properties (e.g., ruggedness). This gives a much richer picture of their relative 
strengths and weaknesses, compared to simply using the "difhculty" of a land- 
scape [sj. 

The rest of the paper is organized as follows: in Section [2] we give a brief 
overview of previous work, before describing our testing methodology in Section 
[3] We then present our experimental results in Section [4] before concluding in 
Section [5] with a discussion of our findings. 



2 Previous work 



The use of algorithms inspired by physical or natural processes is now well- 
established in the field of optimisation As the number of such algorithms 
grows year-on-year, there is a pressing need to better understand their properties, 
in order that practitioners may make informed decisions about which method is 
best-suited to a particular problem, under certain conditions. Although analyti- 



cal methods have been successfully applied to nature-inspired methods 10 11 
their "real world" applicability is not clear, as they often rely on significant 
assumptions and/or simplifications. 

In what follows, we take an experimental approach [12| to studying the se- 
lected algorithms, using an established landscape generation technique [4]. As 
Morgan and Gallagher observe, "In a general sense, an algorithm can be expected 
to perform well if the assumptions that it makes, either explicit or implicit, are 
well-matched to the properties of the search landscape or solution space of a 
given problem or set of problems" (3j . We therefore seek to investigate the per- 
formance of several algorithms on a number of types of fitness landscape with 
specific properties or characteristics. This approach is preferred by Hooker to 
the use of benchmark problems, because the latter "differ in so many respects 
that it is rarely evident why some are harder than others, and they may yet fail 
to vary over parameters that are key determinants of performance. It is better 
generate problems in a controlled fashion... The goal is not to generate realistic 
problems, which random generation cannot do, but to generate several problem 
sets, each of which is homogeneous with respect to characteristics that are likely 



to affect performance" 13 



The fitness landscape approach has been successfully applied to the study of 



various nature-inspired algorithms 14 -16 . Indeed, to our knowledge, landscape 



analysis of nature-inspired algorithms has been largely restricted to evolution- 
ary methods. In this paper we broaden this work considerably, by considering 
several classes of natural algorithms (social, evolutionary and physical). Over- 
all, we study six different nature-inspired methods, as well as stochastic hill- 
climbing as a baseline algorithm. Our empirical approach is informed by previ- 



ous work 17 18 , which emphasises the need to establish a rigorous framework 
for experimental algorithmics. In the next Section, we describe in detail our 
methodology. 



3 Methodology 



3.1 Algorithm selection 

We select, for comparison, a number of nature- inspired algorithms that are com- 
monly applied to continuous function optimisation. These may be classified [To] 
as either social, evolutionary or physical. The social algorithms we select are Bac- 
terial Foraging Optimisation Algorithm (BFOA) 120], Bees Algorithm (BA) [2l] , 
and Particle Swarm Optimisation (PSO) [22]. The evolutionary algorithms se- 
lected are Genetic Algorithms (GA) [23] and Evolution Strategies (ES) [24] , 
and physical algorithms are represented by Harmony Search (HS) [25j. We also 
include random search (RS) and stochastic hill climbing (SHG) as "baseline" 
algorithms. 

We note that the references supplied above for each algorithm may serve sim- 
ply as an example of their application, rather than their precise implementation. 
In terms of implementation, we heed the observation that "Ideally, competing 
algorithms would be coded by the same expert programmer and run on the same 
test problems on the same computer configuration" [12] . With that in mind, we 



use only implementations provided by Brownlee to accompany 26 



3.2 Optimisation problem characteristics 

As Morgan and Gallagher explain [sj, their Max-Set of Gaussians (MSG) method 
[4] is a "randomised landscape generator that specifies test problems as a weighted 
sum of Gaussian functions. By specifying the number of Gaussians and the mean 
and covariance parameters for each component, a variety of test landscape in- 
stances can be generated. The topological properties of the landscapes are intu- 
itively related to (and vary smoothly with) the parameters of the generator." By 
manipulating these parameters, we obtain landscapes with different characteris- 
tics. This allows us to investigate the performance of our selected algorithms on 
landscapes with different features, and to identify which characteristics pose the 
greatest challenge. As Morgan and Gallagher observe, "Different problem types 
have their own characteristics, however it is usually the case that complemen- 
tary insights into algorithm behaviour result from conducting larger experimen- 
tal studies using a variety of different problem types" [sj . We now describe the 
different characteristics (corresponding to problem types) under study in this 
paper. 

Ruggedness of a landscape is often linked to its difficulty [27], and factors 



affecting this include (1) the number of local optima 28 , and (2) ratio of local 



optima to the global optimum [29] 14 . Other significant factors concern (3) 



dimensionality [30| (that is, the number of variables in the objective function), 
(4) boundary constraints (that is, the limits imposed on the value of a variable) 



p3l] , and (5) smoothness of each curve making up the landscape 
of the ranges selected for each characteristic is given in Table 



A summary 



Table 1. A summary of the ranges selected for the characteristics in our fitness space 
(F) 



Characteristic Min Step Max Default 



Number of local optima 





1 


9 


3 


Ratio of local optima 


to 0.1 


0.2 


0.9 


0.5 


global optimum 










Dimensionality 


1 


1 


10 


2 


Boundary constraints 


10 


10 


100 


30 


Smoothness 


10 


10 


100 


15 



3.3 Performance measurement 

In terms of performance metrics, we abstract away from algorithm-specific mea- 
sures, due to the diverse range of methods selected. The following metrics are 
applied: (1) Accuracy: We define this as the average distance from the global 
optimum of the best solution found on a given set of landscape characteristics, 
over a number of runs (that is, the average error). This is the most commonly- 
used assessment metric for optimisation algorithms 4 . The generation tech- 
nique we use creates landscapes with a known global optimum (zero), which 
permits a precise assessment of an algorithm's performance. (2) Diversity: 
One desirable feature of any nature-inspired algorithm is that it maintains a di- 



verse set of solutions 33 . We measure diversity of solutions using the standard 



deviation of the best solutions of all runs on a given set of landscape characteris- 

n 

tics, defined as {:^^ ^^(^i ~ 2;)^) ' (where X is our data set, n is the size of the 

1=1 

data set and x is the mean average). (3) Success rate: We measure this as the 
frequency with which differently-seeded runs of an algorithm are able to find a 



solution within a specified distance from the optimum 34 . We keep the success 
tolerance relatively low (error less than 1.0x10"^) in order to ensure that we 
capture the change in success rate of algorithms which perform poorly. 



3.4 Experimental setup 

In order to generate the landscapes, we used the Matlab code supplied with [i]. 
All landscapes were generated using default parameters of three curves, two di- 
mensions, 0.5 average ratio of local minima to global minimum, 30 units in each 
dimension with a smoothness coefficient of 15), with only the parameter under 
investigation changing for each experiment. We ran each algorithm 100 times 
on each landscape in the set of landscapes generated for each particular char- 
acteristic value (when investigating smoothness, for example, we generated 100 
different landscapes for each value of the characteristic (100 landscapes each for 
smoothness = 10 . . . 100), and ran each algorithm 100 times on each landscape). 



Parameterisation of algorithms provides a significant challenge when evalu- 
ating performance. Our aim is not to perform "competitive testing" jl3j , but 
to establish general performance profiles for different algorithms over different 
types of problem. As such, we use the so-called "vanilla" implementation of each 
algorithm, with general-purpose settings taken from [4| . Where an algorithm has 
a "population size" parameter, we use a value of 50; where an algorithm has a 
"range" or "velocity" parameter, we use a value of 10. A document listing all 
parameter values used is available at the project websitcQ 

Termination criteria were also standardised. The most objective criterion is 
the number of objective function evaluations. This means each algorithm has 
access to the same amount of information from the landscape, and the same 
amount of feedback on potential solutions. Experimentally we determined that 
the selected algorithms generally converged within 20,000 objective function cal- 
culations, so this was used as the termination criterion. 



4 Results 

Space prevents a detailed presentation of full experimental plots, but these are 
available from the project website^. To summarise, we plot the resilience of each 
algorithm to changing landscape characteristics, in the form of a radar plot in 
Figure [TJ To assess the resilience of an algorithm we use the standard deviation 
of the average error across all values of a landscape characteristic, which we 
normalise on a per-characteristic basis. This "ranking" shows which algorithms 
do not show performance variability versus those which are heavily influenced 
by a characteristic. BFOA shows large deviations in average error for boundary 
constraint range, smoothness coefficient changes and dimensionality, indicating 
that BFOA is an algorithm heavily dependent on the landscape of a problem 
- perhaps because of a heavy reliance on careful parameterisation. SHC also 
shows large variance - perhaps, in large part again, to a lack of parameters and 
complicated local optima avoidance techniques. GA and ES show large variation 
with respect to number of local optima, perhaps supporting the argument that 
evolutionary algorithms suffer more than most from the problem of becoming 
"stuck" in local optima. 

All algorithms produce the smallest average error when no local optima 
(minima) are present in the fitness landscape. This is expected, as, with only 
one optimum, there are no alternative solutions to which the algorithms may con- 
verge. We observe the greatest average error with only one optimum from SHC, 
with BFOA (approx. 0.14) also showing a large average error. There are very 
small average errors (almost zero) from GA, ES, PSO, HS, RS and BA. BFOA 
also produces the largest variation in final solutions (0.32). With the introduc- 
tion of only a single local optimum, performance of most algorithms degrades 
significantly. ES and GA suffer significantly, with average error increasing from 
approximately zero to 0.06 and 0.08 respectively, and the standard deviation 
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Fig. 1. Radar plots depicting the standard deviation of the average error of each al- 
gorithm with respect to differing landscape characteristics. Standard deviations are 
normalised on a per-axis basis. Values close to the centre of the plot indicate a larger 
variance in average error, indicating these algorithms are more affected by the charac- 
teristic. In general, the more robust an algorithm, the larger the plot surface area. 



of solutions increasing by around 0.15 for each algorithm. SHC also performs 
poorly, with a similar increase in average error. The least affected are RS (which 
blindly chooses random solutions, and is therefore unaffected by local minima) 
and BA, which contains a global search mechanism. 

For algorithms which do not directly use the gradient of the landscape, we 
would expect to sec no change in their performance as we adjust the ratio 
of local optima parameter. We observe that RS, which selects new solutions 
randomly from the entire search space, offers very similar performance in terms 
of mean error and success rate for all ratio values. Similarly, algorithms which 
perform a global search should be better at avoiding local minima even when 
they are attractive - and this is true for BA and HS. PSO shows little change in 
success rate as the ratio becomes more attractive, owing to the fact that solutions 
are directed towards the best particle, and their own best solution, regardless 
of their individual experience with the gradation of the landscape. Interestingly, 
SHC average error decreases as ratio increases - most likely due to an increased 
availability of 'better' solutions throughout the landscape. ES demonstrates very 
poor, yet consistent, performance as the ratio changes. Success rates are very low, 
and, interestingly, we observe a decrease in the standard deviation of solutions as 
the ratio increases. This suggests that ES is perhaps more "content" to optimise 
at a local minima, with the algorithm getting trapped in these more frequently 
as ratio increases. This could also be true of other algorithms whose deviation 
decrease, such as BFOA and SHC. GA performs in a similar manner to ES with 
regard to average error and diversity, although with a considerably better success 
rate, suggesting that this may be a general problem for algorithms which use an 
evolutionary approach. 

At only one dimension, fitness landscapes are trivially easy. The perfor- 
mance of all algorithms reflects this, with all algorithms performing well on 
landscapes of a single dimension. All algorithms show a success rate (that is, 
optimisation with an error of under l.OxlQ-^) above 90%. As we increase the 
dimensionality to two, most algorithm performances begin to degrade. Suffering 
mostly severely is RS, which is to be expected, as random search is our most ba- 
sic algorithm. Algorithms which also perform poorly at only two dimensions are 
ES, BA and PSO. It is perhaps surprising, at first, to see BA performing poorly, 
given that the algorithm contains a randomly sourced global search. However, 
this global search is effectively RS, which performs poorly, so we can assume 
the global search is not covering enough of the landscape. Coupled with the 
non-adaptive nature of the algorithm (meaning that solution selection around 
the current best area is within a relatively large range), poor algorithm perfor- 
mance is easily explained. We propose that PSO and ES suffer from a similar 
problem, in that exploration is limited, and neither optimise their current best 
as accurately as their adaptive variants. 

Random search exhibits a similar, yet less extreme, reaction to changes in 
boundary constraints as with the increase in dimensionality. This is to be ex- 
pected, as the limit on objective function calculations results in random search 
having less chance to explore the search space. SHC also has an almost linear 



increase in average error, matching the hnear increase in search space size, but 
produces consistently poor results in terms of success. The social system algo- 
rithms (BA and PSO) both exhibit slightly unusual behaviour - as the problem 
space increases, their success rate also increases. This suggests that their reliance 
on a parameter to search within a range is hindering the algorithms when the 
problem space is too small to properly explore. HS provides the best success 
rate for the entire range of sizes we have selected in this problem, indicating 
good exploration of the search space irrespective of the range parameter. BFOA 
also suffers significantly as search space size increases, again implying a heavy 
reliance on the parameter which controls the range of search for new solutions. 
The evolutionary algorithms do not cope particularly well with the increase of 
problem size, with performance in terms of both average error and success rate 
decreasing consistently as size increases. 

The evolutionary algorithms (ES and particularly GA) perform poorly and 
are most affected by changing the smoothness coefficient. BA and PSO all 
also show decreasing success rate as the curves become steeper, as does BFOA 
which relies heavily on gradient information. Harmony search suffers similarly 
to the evolutionary algorithms, and swarm algorithms, as curves become more 
steep. The similarity in terms of success rate for all algorithms suggests that the 
availability of gradient information is something which affects all algorithms. 

5 Conclusions 

In this paper, we have described the results of an extensive study of nature- 
inspired algorithms, in terms of their performance on fitness landscapes with dif- 
ferent characteristics. We studied six nature-based methods (plus two stochastic 
baseline algorithms), varying a number of landscape features. The most signif- 
icant characteristic appears to be the number of local minima, where a combi- 
nation of global and local search appears to be beneficial. On the other hand, 
the ratio of local optima to the global minimum appears to have; little effect on 
the success of the algorithms under study. As expected, dimensionality proved 
problematic for all algorithms, whereas landscape smoothness appeared to have 
httle effect. 

This work offers a contribution to the empirical study of nature-inspired algo- 
rithms, and we hope that it motivates future investigations. To further this work, 
it may be useful to examine a larger collection of nature-inspired algorithms over 
a greater range of values for the characteristics, in order to more fully capture 
a wider variety of algorithmic performance. The current work provides a firm 
foundation for this. 
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